Efficient database management system and method for use therewith

ABSTRACT

A database management system operates by: obtaining a dataset from a memory of the database management system, wherein the dataset includes a set of data records; determining a set of data characteristics of the dataset; determining a list of analytical calculations that are able to be executed on the dataset; determining an analytical calculation characteristic for each analytical calculation of the list of analytical calculations to produce a set of analytical calculation characteristics, wherein the analytical calculation characteristic indicates an estimated execution time to perform the analytical calculation; ranking each analytical calculation of the list of analytical calculations based on the set of data characteristics and the set of analytical calculation characteristics to produce a ranked list of analytical calculations; selecting an analytical calculation from the ranked list of analytical calculations based on the ranking; and executing the selected analytical calculation on the dataset to produce an analytical calculation result.

CROSS REFERENCE TO RELATED APPLICATIONS

The present U.S. Utility patent application claims priority pursuant to35 U.S.C. § 120 as a continuation of U.S. Utility application Ser. No.15/840,558, entitled “EFFICIENT DATABASE MANAGEMENT SYSTEM AND METHODFOR PRIORITIZING ANALYTICAL CALCULATIONS ON DATASETS”, filed Dec. 13,2017, which claims priority pursuant to 35 U.S.C. § 119(e) to U.S.Provisional Application No. 62/433,901, entitled “EFFICIENT DATABASEMANAGEMENT SYSTEMS”, filed Dec. 14, 2016, both of which are herebyincorporated herein by reference in their entirety and made part of thepresent U.S. Utility patent application for all purposes.

U.S. Utility application Ser. No. 15/840,558, is also related to U.S.Provisional Application No. 62/403,231, entitled “HIGHLY PARALLELDATABASE MANAGEMENT SYSTEM,” filed on Oct. 3, 2016, and U.S. ProvisionalApplication No. 62/403,328, entitled “APPLICATION DIRECT ACCESS TONETWORK RDMA MEMORY, filed on Oct. 3, 2016, both of which are herebyincorporated by reference in its entirety.

FIELD OF THE DISCLOSURE

The present invention generally relates to a system and method fororganizing and managing large volume of data, and more particularlyrelates to a massively parallel database management system optimized formanaging time based data. More particularly still, the presentdisclosure relates to a manifest and a silo system in a massivelyparallel database management system, and a method for prioritizinganalytical calculations.

DESCRIPTION OF BACKGROUND

With rapid development and widespread utilization of computertechnologies in the last few decades, large volumes of digital data aregenerated on a daily basis. Organizing and managing such a huge amountof data has promoted the development of database technologies.Relational database management systems (“RDBMS”), such as OracleDatabase Management System, Microsoft SQL Database Management System andMySQL Database Management System, have thus been proposed and gainedbroad acceptance for data management. Such database management systemsstore data by rows of tables. Querying and retrieving data from theconventional databases oftentimes include retrieving a list of recordswhile such records contain information that is not requested. Forexample, the illustrative SQL query causes a conventional databasemanagement system to read all fifty rows from a disk drive storing therows:

select column1 from table1 where key>100 and key<151

In the illustrative SQL query, column1 is a column of a table 1, and keyis another column (such as a primary key) of the table 1. While onlydata in column1 is requested, data in other columns of table1 is alreadyread from a storage disk drive. Furthermore, the conventional databasemanagement systems do not store data in an ordered manner on physicaldisk drives. However, many types of data (such as network logs, networkaccess data, financial transaction data, weather data, etc.) are ofextremely high volume and ordered by time. Accordingly, there is a needfor a highly parallel and efficient database system that is optimizedfor managing large volumes of time based data. There is a further needfor a highly parallel and efficient database system for storing data bycolumns for faster and more efficient data retrieval.

Conventional database management systems typically generate a largenumber of indexes for data. Such indexes logically identify rows (alsoreferred to herein as records). Rows of data within a table are storedon disk drives. Related rows, such as rows of a particular order bytime, are usually not consecutive stored on disk drives. Rows could alsobe related by other factors. Retrieving a set of related records thusinvolves multiple disk reads of data dispersed at different locations ona disk drive. Accordingly, there is a need for a highly parallel andefficient database system for storing related data consecutively ornearby on a disk drive to reduce the number of disk reads in serving adata request, and providing an efficient structure for locating suchdata on a disk drive. There is a further need for the new databasemanagement system to load the structure in memory for higher performancein locating data on disk drives.

To improve data retrieval performance, conventional database managementsystems take advantage of high end hardware platforms, such as acomputer with multiple sockets and a large amount of memory. Each of thesockets includes one or more processing units (also interchangeablyreferred to herein as cores). A processing unit housed in one socket canaccess resources (such as disk drives and memory) local to anothersocket. Such cross socket access incurs a performance penalty due tolatency and bandwidth limitations of the cross-socket interconnect.Accordingly, there is a need for a highly parallel and efficientdatabase management system that improves performance by avoiding thecross socket boundary access. The present disclosure incorporates novelsolutions to overcome the above mentioned shortcomings of conventionaldatabase management systems.

OBJECTS OF THE DISCLOSED SYSTEM, METHOD, AND APPARATUS

Accordingly, it is an object of this disclosure to provide a paralleldatabase management system optimized for managing large volumes of timebased data.

Another object of this disclosure is to provide a parallel databasemanagement system with silo systems that utilize only local resourcesfor faster performance.

Another object of this disclosure is to provide a parallel databasemanagement system with silo systems that utilize only local resources toavoid latency and bandwidth limitations inherent in interconnect access.

Another object of this disclosure is to provide a parallel databasemanagement system with silo systems that utilize only local memory andlocal disk drives for faster performance.

Another object of this disclosure is to provide a parallel databasemanagement system with a signal rich manifest describing physicallocation of data stored on a disk drive for locating the maximum amountof data while taking the least amount of memory and disk space.

Another object of this disclosure is to provide a parallel databasemanagement system with a hierarchical manifest describing physicallocation of data stored on a disk drive for faster data retrieval from astorage disk by direct reads.

Another object of this disclosure is to provide a parallel databasemanagement system with a manifest for each segment.

Another object of this disclosure is to provide a parallel databasemanagement system with a manifest stored in each segment.

Another object of this disclosure is to provide a parallel databasemanagement system with a manifest stored at end of each segment.

Another object of this disclosure is to provide a parallel databasemanagement system with a hierarchical manifest in a physically backedmemory region for faster access minimizing page faults.

Another object of this disclosure is to provide a parallel databasemanagement system with a hierarchical manifest organizing data bycluster keys.

Another object of this disclosure is to provide a parallel databasemanagement system with a hierarchical manifest organizing data by timebased data buckets for each cluster key.

Another object of this disclosure is to provide a parallel databasemanagement system with a hierarchical manifest organizing data by timebased data buckets of equal time frames.

Another object of this disclosure is to provide a parallel databasemanagement system with time based data stored on disk drives based onthe order of time stamps of the data.

Another object of this disclosure is to provide a parallel databasemanagement system with time based data of different time periods storedin different segment groups.

Another object of this disclosure is to provide a parallel databasemanagement system with data records stored by columns for fasterperformance in retrieving data from physical disk drives.

Another object of this disclosure is to provide a parallel databasemanagement system with data records stored by columns for reducing readsof physical disk drives in data retrieval.

Another object of this disclosure is to provide a parallel databasemanagement system with data records stored by columns in coding blocksof different coding lines on a segment to allow fewer reads in dataretrieval.

Another object of this disclosure is to provide a parallel databasemanagement system to store data records by columns in a segment with amanifest indicating the location of the data on the physical disk driveof the segment for faster data retrieval.

Another object of this disclosure is to provide a parallel databasemanagement system to store a data record along with a confidence aboutthe accuracy of the data record.

Another object of this disclosure is to provide a parallel databasemanagement system that prioritizes analytical calculations on largedatasets.

Another object of this disclosure is to provide a parallel databasemanagement system that prioritizes analytical calculations on largedatasets based on characteristics of the analytical calculations andcharacteristics of the dataset.

Another object of this disclosure is to provide a parallel databasemanagement system that prioritizes an analytical calculation based therank of a similar analytical calculation based on characteristics of thetwo analytical calculations.

Other advantages of this disclosure will be clear to a person ofordinary skill in the art. It should be understood, however, that asystem or method could practice the disclosure while not achieving allof the enumerated advantages, and that the protected disclosure isdefined by the claims.

SUMMARY OF THE DISCLOSURE

Generally speaking, pursuant to the various embodiments, the presentdisclosure provides a massively parallel database management system formanaging massive volumes of data. In particular, massively paralleldatabase management system is optimized for managing time based data. Adatabase management software application running in user space directlyaccesses a disk drive to store and retrieve data for faster performance.The time based data is stored in coding blocks of segments of a segmentgroup of a cluster. Coding blocks of different segments within the samesegment group are grouped in coding lines. The cluster includes a set ofnodes, each of which includes one or more storage disk drives. Each diskdrive includes one or more segments.

Each node includes one or more sockets while each socket houses a set(meaning one or more) of processing units. A socket and its processingunits are operatively coupled to a set of local resources, such as alocal memory, a local disk drive and a local network interface card. Aprocessing unit accesses the local devices at a higher speed thanaccesses to remote devices that are local to a different socket. The twosockets are interconnected by an interconnect interface. The crosssocket access is slower due to latency and bandwidth limitation in theinterconnect interface. A socket, the process units housed in thesocket, physical devices local to the socket and an operating systemrunning on the processing units are termed herein as a silo. Themassively parallel database management system implementing the silooriented computing achieves faster performance due to the fact that thedata management processing within different silos uses only localdevices. Data management threads are pinned to specific processing unitswith a silo such that the threads only access local memory and otherlocal resources.

For improved performance in data retrieval reads and data storingwrites, the novel data management system accesses disk drives directlywithout going through middle layers, such as a file system of anoperating system. The data management system software applicationmaintains a manifest to track the exact physical location where aparticular piece of data is stored in a segment of a physical diskdrive. The manifest embodies a compact structure such that it minimizesstorage overhead for relational information in a segment while occupyinga small footprint. The manifest is thus optimized to occupy less memoryand disk drive space while providing the maximum amount of signal. Themanifest is stored, for example, at the end of each segment while datais stored in coding blocks from the beginning of the segment.

Since data requests usually demand data in certain columns, but not allcolumns of data records, the database management system softwareapplication further improves on conventional technologies by storingdata by columns in coding blocks on segments within a cluster. Byretrieving only data of one or multiple columns, the number of reads isreduced because the amount of data read is less that the total amount ofdata within all of the relevant records. To further speed up dataqueries, different segment groups store time based data of differenttime periods. In such a case, a requested data record is first quicklynarrowed to a segment group based on the time stamp of the data record.

The manifest indicates the location of data stored in the correspondingsegment. The manifest organizes data records by cluster keys. For eachcluster key, data is organized as data buckets of sequential, but notnecessarily contiguous, time periods. The different time periods are ofthe same time duration (also referred to herein as time frame) in oneimplementation. For each data bucket, data is stored by columns, whereineach stored column is indicated by coding lines and storage byteoffsets.

Further in accordance with various embodiments, the present teachingsprovide a database management system that stores data records along withconfidence data. The confidence data indicates a confidence in theaccuracy of a data record or data point. In addition, analyticalcalculations for analyzing large datasets are prioritized for effectivedata analysis. The prioritization can be based on characteristics of theanalytical calculations and/or characteristics of a particular dataset.Furthermore, a rank of one analytical calculation is assigned to asimilar analytical calculation. The ranks are determined based on, forexample, execution results of the analytical calculations on thedataset.

Further in accordance with the present teachings is a method forprioritizing analytical calculations on a dataset stored in a largedatabase. The method includes retrieving the dataset from the largedatabase. The dataset includes a set of data records. The method furtherincludes determining a set of data characteristics of the dataset,determining a list of analytical calculations, prioritizing the list ofanalytical calculations based on the set of data characteristics and aset of analytical calculation characteristics of the list of analyticalcalculations to associate a rank to each analytical calculation withinthe list, selecting an analytical calculation from the list based theranks, and executing the selected analytical calculation on the dataset.The set of data characteristics includes a number of data records withinthe dataset. The set of data characteristics also includes a data typeof a column of each data record within the dataset. The list ofanalytical calculations further includes an analytical calculation thatdetermines an abnormality of data records within the dataset. Theabnormality is an error, data distortion, or data noise. The list ofanalytical calculations includes a first analytical calculation and asecond analytical calculation. Execution of the first analyticalcalculation on the dataset is faster than execution of the secondanalytical calculation on the dataset.

Further in accordance with the present teachings is a method forprioritizing analytical calculations on datasets stored in a largedatabase. The method includes determining a list of analyticalcalculations, and retrieving a dataset from a large database. Thedataset includes a set of data records. The method also includesexecuting each analytical calculation within the list of analyticalcalculations on the dataset to determine a result score to form acorresponding list of result scores. In addition, the method includesdetermining a rank for each analytical calculation within the list ofanalytical calculations based on the list of result scores, andcomparing a first analytical calculation with analytical calculationswithin the list of analytical calculations to determine that the firstanalytical calculation is similar to a second analytical calculationwithin the list of analytical calculations. The first analyticalcalculation is not in the list of analytical calculations. The methodfurther includes assigning the rank of the second analytical calculationto the first analytical calculation. The first analytical calculation isdetermined to be similar to the second analytical calculation based oncharacteristics of the first and second analytical calculations.

BRIEF DESCRIPTION OF THE DRAWINGS

Although the characteristic features of this disclosure will beparticularly pointed out in the claims, the invention itself, and themanner in which it may be made and used, may be better understood byreferring to the following description taken in connection with theaccompanying drawings forming a part hereof, wherein like referencenumerals refer to like parts throughout the several views and in which:

FIG. 1 is a simplified block diagram of a node within a cluster of ahighly parallel database management system in accordance with thisdisclosure.

FIG. 2A is a flow chart illustrating a process by which a node within acluster of a highly parallel database management system implements silooriented resource accesses in accordance with this disclosure.

FIG. 2B is a flow chart illustrating a process by which a node within acluster of a highly parallel database management system implements silooriented resource accesses in accordance with this disclosure.

FIG. 2C is a flow chart illustrating a process by which a node within acluster of a highly parallel database management system implements silooriented resource accesses in accordance with this disclosure.

FIG. 3 is a simplified block diagram depicting a segment with a manifestin accordance with this disclosure.

FIG. 4 is a table illustrating time stamp based data in accordance withthis disclosure.

FIG. 5 is a simplified block diagram illustrating the layout of recordsstored in a segment in accordance with this disclosure.

FIG. 6 is a simplified block diagram illustrating a storage cluster oftime based data in accordance with this disclosure.

FIG. 7 is a simplified block diagram illustrating a logicalrepresentation of a manifest in accordance with this disclosure.

FIG. 8 is a simplified block diagram illustrating the memory structureof a manifest 700 in accordance with the teachings of this disclosure.

FIG. 9 is a data record without 100% confidence of accuracy inaccordance with the teachings of this disclosure.

FIG. 10A is a flowchart depicting a process by which a computerprioritizes analytical calculations in accordance with the teachings ofthis disclosure.

FIG. 10B is a flowchart depicting a process by which a computerprioritizes analytical calculations in accordance with the teachings ofthis disclosure.

A person of ordinary skills in the art will appreciate that elements ofthe figures above are illustrated for simplicity and clarity, and arenot necessarily drawn to scale. The dimensions of some elements in thefigures may have been exaggerated relative to other elements to helpunderstanding of the present teachings. Furthermore, a particular orderin which certain elements, parts, components, modules, steps, actions,events and/or processes are described or illustrated may not be actuallyrequired. A person of ordinary skill in the art will appreciate that,for the purpose of simplicity and clarity of illustration, some commonlyknown and well-understood elements that are useful and/or necessary in acommercially feasible embodiment may not be depicted in order to providea clear view of various embodiments in accordance with the presentteachings.

DETAILED DESCRIPTION

Turning to the Figures and to FIG. 1 in particular, a simplified blockdiagram of a node within a cluster of a highly parallel databasemanagement system is shown and generally indicated at 100. The databasestorage node 100 includes two sockets 106 and 108, each of whichincludes one or more processing units (also interchangeably referred toherein as cores and central processing units). The node 100 alsoincludes a memory (such as 32 GB of DRAM) 110, a storage disk drive 114,and a networking interface (“NIC”) 118 that are operatively coupled tothe socket 106. An operating system (such as Linux operating system) 122runs on the processing units of the socket 106. The memory 110, thesocket 106, the NIC 118 and the disk drive 114 are collectively referredto herein as a silo 102. The silo system 102 includes all processingunits within the socket 106 and all the disk drives (such as the diskdrive 114) operatively coupled to the socket 106. The node 100 furtherincludes a memory 112, two storage disk drives 115 and 116, and a NIC120 that are operatively coupled to the socket 108. The memory 112, thestorage disk drives 115-116, and the NIC 120 are collectively referredto herein as a silo 104.

A specialized computer software 126 for managing data runs on theoperating system 122. In one implementation, the operating system 122 isa single instance running on the sockets 106-108 of the node 100. In oneimplementation, the specialized computer software 126 programs each siloto perform a part of a task. The specialized computer software 126 canalso program one silo (such as the silo 102) to perform one task, andanother silo (such as the silo 104) to perform a different task.

The disk drives 114-116 are storage devices for storing data, and canbe, for example, Non-volatile Random-Access Memory (“NVRAM”), SerialAdvanced Technology Attachment (“SATA”) Solid State Drives (“SSDs”), orNon-volatile Memory Express (“NVMe”). As used herein, drives, storagedrives, disk drives and storage disk drives are interchangeably used torefer to any types of data storage devices, such as NVRAM, SATA, SATASSDs and NVMe. Each of the disk drives (such as the drives 114-116) hasone or more segments. For ease of illustration, each of the disk drives114-116 is said to include only one segment and interchangeably referredto as a segment herein. Segments within a cluster form a segment group.

The processing units within the socket 106 directly access the memory110, the NIC 118 and the disk drive 114 over electrical interfaces, suchas Peripheral Component Interconnect Express (“PCIe”). For example, thesocket 106 directly accesses these physical devices via a PCIe bus, amemory control, etc. Similarly, the socket 108 directly access thememory 112, the NIC 120 and the disk drives 115-116.

In contrast, the processing unit(s) within the socket 108 accesses thememory 110, the disk drive 114 and the NIC 118 via an interconnectioninterface 152. Similarly, the processing unit(s) within the socket 106accesses the NIC 120, the disk drives 115-116 and the memory 112 via thesame interconnection interface 152. The access over the interconnectioninterface 152 between the sockets 106 and 108 is referred to herein asan indirect connection. In other words, a socket within each silodirectly accesses physical devices within the same silo, and indirectlyaccesses physical devices within a different silo. Physical deviceswithin one silo are said to be local to the silo and remote to adifferent silo.

In one implementation, the interface 152 is a QuickPath Interconnect(“QPI”) interface or an UltraPath Interconnect (“UPI”) interface. Theindirect access between the silos 102-104 incurs a performance penaltydue to latency inherent in indirect access. Furthermore, theinterconnect interface 152 becomes a bottleneck in indirect access. Inaddition, the interconnect interface 152 has a bandwidth limitation.Accordingly, accessing remote devices over the interconnect interface152 is less desirable. To overcome the performance issues imposed by theindirect access, the present teachings provide the specialized databasemanagement system software 126 to implement a silo oriented databasesystem.

In the silo based data management system, the instance of thespecialized database management system software 126, running on theprocessing unit(s) within the socket 106, accesses only the localresources, such as the memory 110, the NIC 118 and the disk drive 114that are local to the socket 106 and all the processing units within thesocket 106. Similarly, the instance of the software 126 running on theprocessing unit(s) within the socket 108 accesses only the NIC 120, thememory 112 and the disk drives 115-116 local to the socket 108 and allthe processing units within the socket 108. In other words, the instanceof the software 126 running on the socket 108 do not access the remotelyconnected physical devices 110, 114, 118 when, for example, data queriesare performed. However, cross-silo access is possible in certain cases,such as system startup, shutdown and administrative actions. Forinstance, performance polling is an administrative action. It should benoted that the silo boundary based computing is programmed for a set ofpredetermined functionality. For example, for storing data into andretrieving data from a database and disk drives, the specialized program126 limits its access to local devices and avoids remote access to adifferent silo. The silo boundary control is further illustrated byreference to FIGS. 2A, 2B and 2C.

Referring to FIGS. 2A, 2B and 2C, three flow charts illustratingprocesses by which the node 100 implements the silo oriented highlyefficient database management are shown and generally indicated at 200A,200B and 200C respectively. The processes 200A-200C are performed by thespecialized database management program 126. The process 200A isinitiated when the program 126 is loaded and run by the processing unitswithin a socket of a silo, such as the socket 106 of the silo system102. In one implementation, the software program 126 runs as a processin the silo 102. The process includes one or more threads. The threadswithin the process share the same virtual address space and can allaccess the same physical resources (such as memory and other physicaldevices). At 202, the specialized database management software 126determines the identification of a list of local devices, such as theprocessing units within the socket 106, the memory 110, the disk drive114 and the NIC 118. For instance, the software 126 queries theoperating system 122 for identification and other information of thelist of local devices. Each physical device within the list can beidentified by, for example, a name or a handle.

At 204, the special software program 126 performs a specialized memoryallocation to allocate a huge page of the memory 110. The huge page is abig swatch of memory (such as 1 GB) that is a virtual memory region. Thehuge page is physically backed by the memory 110. In other words, thevirtual memory region corresponds to a region of the same size on thememory device 110. Multiple accesses to the virtual memory region resultin the same physical region being accessed. A processor maintains acache of virtual-to-physical page mappings (i.e., the TranslationLookaside Buffer (“TLB”)); and by utilizing a huge page the specialsoftware is able to address larger regions of memory with fewer TLBcache entries. The physically backed huge page is also referred toherein as a physical huge page of memory. The physically backed hugepage is within the silo boundary, and corresponds to a segment manifest.

At 206, the specialized software program 126 loads a segment manifestinto the physically backed huge page. The manifest describes ahierarchical structure indicating the location of data in the segment(such as the disk drive 114). In one implementation, each segment storesa manifest. A segment with a manifest is further illustrated byreference to FIG. 3.

Turning to FIG. 3, a simplified block diagram depicting a segment 114with a manifest 302 is shown. In the segment 114, data is stored incoding blocks, such as the coding blocks 312-318. Coding blocks arewritten into the segment 114 in a sequential order starting from thebeginning of the segment 114. In one implementation, the manifest 302 ofthe segment 114 is stored at the end of the segment 114. In oneembodiment, the manifest 302 occupies a fixed size of the disk space onthe segment 114. As further described below, the manifest 302 containsthe maximum amount of signal for a certain size of storage. The signalis data indicating information about other data, such as the physicallocation of a block of data within a storage drive.

Returning to FIG. 2A, the manifest resides in memory for the bestperformance in locating data stored in the local disk drive 114. At 208,the specialized database software 126 pins a thread within the processof the software 126 to one or more processing units (such as CPUs andcores) within the socket 106 via operating systems calls. For example,the calls include “pthread_setaffinity_np” and/or “sched_setaffinity” ona Linux operating system. Operations (such as searches) on the loadedmanifest that are performed by the pinned thread are then only performedon the memory 110 within the silo 102, not the memory 112 that is remoteto the silo 102.

Referring to FIG. 2B, at 222, the specialized database managementsoftware 126 receives a chunk of data for storing into the disk drive114 via the NIC 118. The chunk of data is some amount of data, such as aset of time based data records of the same or different cluster keys. At224, the pinned thread processes the chunk of data for storing it ontothe disk drive 114. For example, the pinned thread places the chunk ofdata into an open coding block, and updates the manifest to reflect theexact location where the chunk of data is stored in the segment 114.When the open coding block is full, at 226, the pinned thread directlyflushes the coding block into the segment 114. It should be noted thatthe updated manifest is also flushed to the segment 114 periodically orwhen certain events occur.

Referring to FIG. 2C, at 242, the specialized database managementsoftware 126 receives a request for a chunk of data, such as a set ofcolumns of certain records. At 244, the pinned thread searches themanifest in the physically backed huge page to determine the location ofthe coding blocks containing the requested data in the segment 114. At246, the pinned thread reads the coding blocks from the segment 114. At248, the pinned thread returns the request chunk of data over the NIC118. It should be noted that the processes 200B-200C directly accessesthe disk drive 114 using its identification determined by the process200A. Furthermore, the process 200C directly accesses the NIC 118 usingits identification determined by the process 200A. Accordingly, theoperations to store a chunk of data are performed within a single silo;and the operations for retrieving and returning a chunk of data are alsoperformed within a single silo. The silo oriented database managementthus provides superior performance and efficiency.

Many types of data are generated in great volumes and of similar or sameformats. For example, a computer network logger produces large volumesof records of the same format. Each record includes a time stamp(meaning the time when the record is generated), a cluster key, and anumber columns of other types of data. The cluster key can identify, forinstance in network log data, a source IP address and a destination IPaddress. The source IP address is the IP address of the computer ordevice sending the data contained in the record, while the destinationIP address is the IP address of the computer or device receiving thedata contained in the record. Another example of the time based data isweather data. Such time stamp based data is uploaded to a databasemanagement system to be stored in disk drives, such as the disk drives114-116. A logical representation of the time based data is furtherillustrated by reference to FIG. 4.

Referring to FIG. 4, a table illustrating time stamp based data is shownand generally indicated at 400. The data is represented as a list ofrecords 0-M (M stands for a positive integer). Each record has a timestamp in column 0, such as Oct. 12, 2016, 19:03:01, CST. The time stampmay further include additional information, such as milliseconds. A timestamp can also be represented by an integer, instead of a text string.Column 1 of the table 400 contains the cluster key of each record.Columns 2 through N (N stands for a positive integer) contain other dataof each record.

The records with the same cluster key are said to be related. Taking anetwork logger as an example, the cluster key is the pair of source IPaddress and the destination IP address. All records with the samecluster key are data sent from a particular computer or device toanother particular computer or device, and are said to be relatedherein. The related records have different time stamps and are alsoordered by the time stamps. For instance, records 0-500 have a samecluster key while records 501-1000 share a different cluster key.

To maximize the performance in serving requests for such data after itis stored on the disk drives 114-116, the present database managementsystem stores the records 0-M based on columns, instead of rows. Dataqueries usually request one or more columns of certain records, such asrecords during a particular time period. Storing the records 0-M bycolumns allows the minimum amount of reads to retrieve the desired datafrom a disk drive. The column based data storage in the highly paralleldatabase management system is further illustrated by reference to FIG.5.

Referring to FIG. 5, a simplified block diagram illustrating the layoutof records stored in the segment 114 is shown. A set of representativecoding blocks of data are indicated at 502 through 538 with the codingblocks 506, 512, 518, 524, 530, 536 being the parity blocks storingparity information for the corresponding coding lines. Each coding blockof the coding blocks 502-538 is associated with a coding line thatencompasses all segments within a segment group.

For example, data of Column 0 of the records with cluster key 0 (meaninga first cluster key) during a particular time period is stored in codingblock 502; data of column 1 of the records with cluster key 0 during theparticular time period is stored in coding blocks 502-504; data ofcolumn 2 of the records with cluster key 0 during the particular timeperiod is stored in coding blocks 504, 508-510; data of column 3 of therecords with cluster key 0 during the particular time period is storedin coding blocks 510 and 514; data of column 4 of the records withcluster key 0 during the particular time period is stored in codingblocks 514-516, 520-522, 526; data of column 0 of the records withcluster key 1 during the particular time period is stored in codingblock 526; data of column 1 of the records with cluster key 1 during theparticular time period is stored in coding blocks 526-528; etc. Recordsof the cluster key 0 (as well as the cluster key 1) during theparticular time period are ordered by their corresponding time stampsfrom, for example, the oldest to the newest.

The time based data is sequentially stored in segments groups, each ofwhich comprises a set of segments. A particular time period is mapped toa small fixed set of segment groups. For example, in one implementation,a particular time period is mapped to a unique segment group. As anadditional example, a particular time period is mapped to two segmentgroups in a different implementation due to the fact that segment groupscan overlap slightly in time at their boundaries. The mapping is furtherillustrated by reference to FIG. 6. Turning to FIG. 6, a simplifiedblock diagram illustrating a storage cluster of time based data is shownand generally indicated at 600. The cluster 600 includes a set of nodes,of which two are indicated at 602 and 604. The node 602 includes datastorage disk drives 606 (such as the drive 114), 608 and 610 while thenode 604 includes disk drives 612, 614 and 616. The drive 606 includes asegment 622; the drive 608 includes three segments 624, 626 and 628; thedrive 610 includes two segments 630 and 632; the drive 612 includes asegment 642; the drive 614 includes three segments 644, 646 and 648; andthe drive 616 includes two segments 650 and 652. The illustrativecluster 600 includes segment groups 672, 674, 676, 678, 680 and 682. Thesegment group 672 includes the segments 622, 642 and other segments (notshown). As another example, the segment group 680 includes the segments630 and 650.

The time based data between time TA and time TB is stored in the segmentgroup 672; the time based data between time TB and time TC is stored inthe segment group 674; the time based data between time TC and time TDis stored in the segment group 676; and so on. The time stamps TA, TB,TC, TD, TE, TF and TG are ordered from the oldest to the latest.Accordingly, when a data record is requested, the segment group storingthe record is first determined based on the time stamp of the record.The time based storage of data in the cluster 600 thus provides anefficient and faster response to a data query. The lengths of differenttime periods, such as from TA to TB and from TB to TC, may differ.

When time based data records are received, a segment group and a segmentwithin the segment group is first determined for storing the record. Forexample, a function is performed on the cluster key of the records todetermine the segment group and the segment. The function is shownbelow:

function (cluster key)=segment group identifier and segment identifier

The data records are then forwarded to the node (such as the node 100)having the segment. The data records are then received by the targetnode. For example, the data record is received at 222 of the process200B. The function (cluster key) enables even distribution data recordsbetween segments within a segment group.

For efficiently placing and searching the time based data records, ahierarchical manifest for each segment is created and managed by thespecialized database management software 126. The manifest is furtherillustrated by reference to FIGS. 7 and 8. Turning first to FIG. 7, alogical representation of a manifest is shown and generally indicated at700. Time based data is grouped by cluster keys (such as the cluster key0 and the cluster key 1); and time based data of each cluster key isgrouped into buckets based on time. For example, a first data bucket ofthe cluster key 0 includes data from time stamp TA1 to time stamp TA2; asecond data bucket includes data from time stamp TA2 to time stamp TA3;and a third data bucket includes data from time stamp TA3 to time stampTA4. In one implementation, the time period for each data bucket is thesame. In other words, TA2−TA1=TA3−TA2=TA4−TA3.

Within each data bucket, data records are organized by columns startingfrom column 0 to column 1 to column 2, and so on. Taking the cluster key0 as an example, the data in the column 0 within the bucket of theperiod from TA1 to TA2 is stored in one or more coding blocks. Thecoding blocks are identified by a starting coding block number SL0, andan ending coding block number EL0. The coding block numbers SL0 and EL0are also referred to herein as a starting coding block line and anending coding block line. Accordingly, SL0 and EL0 identify one or moreconsecutive blocks on the segment storing the corresponding data. SB0indicates the starting byte location from the beginning of the firstcoding block of the one or more consecutive coding blocks, while EB0indicates the ending byte location from the beginning of the firstcoding block of the one or more consecutive blocks. In other words, thestorage space starting from the byte at SB0 to the byte at EB0 in theone or more consecutive coding blocks store the data of the column 0 ofthe time based records in the data bucket between TA1 and TA2 of thecluster key 0. A data bucket cannot be empty. If no data is present fora particular time period, no bucket is stored, and during retrieval thelack of a bucket is interpreted as there being no data for that timeperiod. In one embodiment, the manifest is immutable; and, if changesare required, the entire manifest is regenerated.

Referring to FIG. 8, a simplified block diagram illustrating the memorystructure of the manifest 700 is shown. Cluster keys are stored inmemory slots 802, 804 and 806 (indicating multiple memory slots). Eachof these slot further stores a location, such as offset from thebeginning of the manifest 700, of the corresponding buckets for theassociated cluster key. Taking cluster key 0 in the memory slot 802 asan example, the data bucket location information is pointed to by thelocation and stored in the memory slots 808, 810 and 812. Taking thefirst data bucket as an example, it is indicated in the memory slot 808,which contains the time stamps of the bucket and a location pointing tothe column information of the bucket. The location points to the memoryslot 822, which stores information (such as data type) of the column 0and a location pointing to the memory slot 842. The memory slot 842stores the starting coding line number (SL0), the ending coding linenumber (EL0), the starting byte offset (SB0) and the ending byte offsetEB0. There could be more than one memory slot (such as the memory slot842) corresponding to a particular column when the span of data for akey/column pair intersects with one or more parity blocks. In such acase, more than one memory slot is required to store the multiplenon-contiguous sub-spans. The compact structure of the manifest 700contains the maximum amount of signal about stored data while using theleast amount of memory.

In one embodiment, the time based data is compressed before it is storedinto a segment of the node 100. For instance, the data of column 3 of aparticular data bucket of a particular cluster key is encoded. Thecompression can be optionally performed on some columns. For example,the compression is not performed on the time stamp and cluster keycolumns. The compression form can be, for example, Run-Length Encoding(“RLE”). In one implementation, the compression is performed at 224 ofthe process 200B.

Certain types of data, such as genomic base pairs in a genome sequence,are created in such a manner that the data value is not known to be 100%accurate. In other words, there is not a 100% confidence in the accuracyof such data. For instance, a gene sequencer may estimate that a genomicbase pair at a given location is 90% likely to be C-G and 10% likely tobe A-T. As an additional example, when network traffic data iscollected, the accuracy of each data record may be affected by the biterror rate of the network hardware or some other reasons. Whenmathematical and statistical analysis is later performed on such datawithout 100% confidence in its accuracy, the confidence of thecalculated output data would be affected by the less than 100%confidence in the network traffic data. Accordingly, in one embodiment,the confidence information about the data is stored in the database.When the data records are retrieved from the database system storingsuch records, the corresponding data confidence is also retrieved. Thedata confidence is further incorporated and considered in the analysisof the data records.

The data without 100% confidence in accuracy and the confidenceinformation are further illustrated by reference to FIG. 9. Referring toFIG. 9, a data record without 100% confidence in its accuracy is shownand generally indicated at 900. The data record 900 includes columns 902through 908. The confidence information is stored in one or moreadditional columns, such as the column 910. The data record 900 is firstconstructed in memory of a database management system computer and thenstored in, for example, the segment 114.

Various datasets, such as network traffic data, financial transactions,and digital sensor data, are growing rapidly each day and becoming solarge that humans can no longer examine such data and get a sense ofwhat is unusual with such datasets. Accordingly, computers are needed toanalyze these large datasets to determine whether any data abnormalityare present. Computers generally analyze a dataset by performinganalyses, such as calculating a standard deviation or a distance betweendata points. As used herein an analysis is also referred to as acalculation. On a large dataset, only a limited number of calculationscould be effectively performed. Accordingly, prioritizing calculationsto perform on large datasets is more desirable.

For example, it is beneficial to prioritize those next calculations ofdata abnormality in a dataset by prioritizing the calculations likely tocomplete faster. In a different implementation, future analyticalcalculations are prioritized based on how the results of previouscalculations are scored. An analytical calculation similar to apreviously executed calculation with high scoring results is alsoprioritized higher. In other words, the analytical calculation isassigned with the same priority score. The analytical calculationprioritization is further illustrated by reference to FIGS. 10A and 10B.

Referring to FIGS. 10A and 10B, two flowcharts depicting two processesby which a computer (such as the node 100) prioritizes analyticalcalculations are shown and generally indicated at 1000A and 1000Brespectively. At 1002, a specialized software application running on thecomputer determines characteristics of a dataset. The characteristicsinclude, for example, the number of records in the dataset and datatypes of columns of the records. At 1004, the software applicationdetermines a list of analytical calculations that may be executed on thedataset for determining any abnormality (such as errors, distorted data,data noise, etc.) in the dataset. At 1006, the software applicationprioritizes the list of analytical calculations based on thecharacteristics of a dataset and the characteristics of each analyticalcalculation in the list. For example, whether a calculation processes adataset by performing only linear operations (such as comparisons) is acharacteristic of the analytical calculation. As an additional example,whether a calculation processes a dataset by performing square rootoperations is a characteristic of the analytical calculation. Somecharacteristics deem the execution of a calculation on a dataset to beslower while others are faster. The prioritization associates a rankwith each analytical calculation. At 1008, the software applicationselects the highest ranked analytical calculation from the prioritizedlist of analytical calculations. At 1010, the software applicationexecutes the selected analytical calculation on the dataset.

Referring now to FIG. 10B, at 1062, the software application executeseach analytical calculation within a list of analytical calculations ona particular dataset. The list of analytical calculations includes oneor more calculations. At 1064, the software application determines aresult score of the execution of each analytical calculation on thedataset. At 1066, the software application ranks the list based on thescores. At 1068, the software application determines than an unlistedanalytical calculation (meaning an analytical calculation that is not inthe list) is similar to a listed analytical calculation (meaning aparticular analytical calculation within the list). For example, thesimilarity is based on similar characteristics of the two analyticalcalculations. At 1070, the software application associates the rank ofthe listed analytical calculation with the unlisted analyticalcalculation.

Obviously, many additional modifications and variations of the presentdisclosure are possible in light of the above teachings. Thus, it is tobe understood that, within the scope of the appended claims, thedisclosure may be practiced otherwise than is specifically describedabove.

The foregoing description of the disclosure has been presented forpurposes of illustration and description, and is not intended to beexhaustive or to limit the disclosure to the precise form disclosed. Thedescription was selected to best explain the principles of the presentteachings and practical application of these principles to enable othersskilled in the art to best utilize the disclosure in various embodimentsand various modifications as are suited to the particular usecontemplated. It should be recognized that the words “a” or “an” areintended to include both the singular and the plural. Conversely, anyreference to plural elements shall, where appropriate, include thesingular.

It is intended that the scope of the disclosure not be limited by thespecification, but be defined by the claims set forth below. Inaddition, although narrow claims may be presented below, it should berecognized that the scope of this invention is much broader thanpresented by the claim(s). It is intended that broader claims will besubmitted in one or more applications that claim the benefit of priorityfrom this application. Insofar as the description above and theaccompanying drawings disclose additional subject matter that is notwithin the scope of the claim or claims below, the additional inventionsare not dedicated to the public and the right to file one or moreapplications to claim such additional inventions is reserved.

What is claimed is:
 1. A method for execution by a computer of adatabase management system, the method comprising: obtaining a datasetfrom a memory of the database management system, wherein the datasetincludes a set of data records; determining a set of datacharacteristics of the dataset; determining a list of analyticalcalculations that are able to be executed on the dataset; determining ananalytical calculation characteristic for each analytical calculation ofthe list of analytical calculations to produce a set of analyticalcalculation characteristics, wherein the analytical calculationcharacteristic indicates an estimated execution time to perform theanalytical calculation; ranking each analytical calculation of the listof analytical calculations based on the set of data characteristics andthe set of analytical calculation characteristics to produce a rankedlist of analytical calculations; selecting an analytical calculationfrom the ranked list of analytical calculations based on the ranking;and executing the selected analytical calculation on the dataset toproduce an analytical calculation result.
 2. The method of claim 1,wherein a data characteristic of the set of data characteristics furtherindicates a number of data records within the dataset.
 3. The method ofclaim 1, wherein a data characteristic of the set of datacharacteristics further indicates a data type of a column of each datarecord within the dataset.
 4. The method of claim 1, wherein theanalytical calculation result indicates an abnormality of data recordswithin the dataset.
 5. The method of claim 4, wherein the abnormality isone of: an error; data distortion; or data noise.
 6. The method of claim1 further comprises: determining to perform another analyticalcalculation on the dataset, wherein the other analytical calculation isnot included in the list of analytical calculations; determining anotheranalytical calculation characteristic for the other analyticalcalculation; comparing the other analytical calculation characteristicto the set of analytical calculation characteristics; when the otheranalytical calculation characteristic matches a first analyticalcalculation characteristic of the set of analytical calculationcharacteristics: determining the ranking of the analytical calculationassociated with the first analytical calculation characteristic;assigning the ranking of the analytical calculation to the otheranalytical calculation; and executing the other analytical calculationon the dataset in accordance with the ranked list of analyticalcalculations.
 7. The method of claim 1, further comprising: executingeach analytical calculation within the list of analytical calculationson the dataset to produce a list of result scores, wherein the list ofresult scores includes a result score for each analytical calculation.8. The method of claim 7, further comprising: updating the ranking ofthe list of analytical calculations based on the list of result scores.9. The method of claim 1, wherein the analytical calculationcharacteristic further includes: a type of calculation to be performedby the analytical calculation, wherein the type of calculation includesone or more of a linear operation and a square root operation.
 10. Themethod of claim 1, wherein the dataset includes a data confidence value.11. A database management system that includes: a memory; a computerthat performs operations that include: obtaining a dataset from thememory of the database management system, wherein the dataset includes aset of data records; determining a set of data characteristics of thedataset; determining a list of analytical calculations that are able tobe executed on the dataset; determining an analytical calculationcharacteristic for each analytical calculation of the list of analyticalcalculations to produce a set of analytical calculation characteristics,wherein the analytical calculation characteristic indicates an estimatedexecution time to perform the analytical calculation; ranking eachanalytical calculation of the list of analytical calculations based onthe set of data characteristics and the set of analytical calculationcharacteristics to produce a ranked list of analytical calculations;selecting an analytical calculation from the ranked list of analyticalcalculations based on the ranking; and executing the selected analyticalcalculation on the dataset to produce an analytical calculation result.12. The database management system of claim 11, wherein a datacharacteristic of the set of data characteristics further indicates anumber of data records within the dataset.
 13. The database managementsystem of claim 11, wherein a data characteristic of the set of datacharacteristics further indicate a data type of a column of each datarecord within the dataset.
 14. The database management system of claim11, wherein the analytical calculation result indicates an abnormalityof data records within the dataset.
 15. The database management systemof claim 14, wherein the abnormality is one of: an error; datadistortion; or data noise.
 16. The database management system of claim11, wherein the operations further include: determining to performanother analytical calculation on the dataset, wherein the otheranalytical calculation is not included in the list of analyticalcalculations; determining another analytical calculation characteristicfor the other analytical calculation; comparing the other analyticalcalculation characteristic to the set of analytical calculationcharacteristics; when the other analytical calculation characteristicmatches a first analytical calculation characteristic of the set ofanalytical calculation characteristics: determining the ranking of theanalytical calculation associated with the first analytical calculationcharacteristic; assigning the ranking of the analytical calculation tothe other analytical calculation; and executing the other analyticalcalculation on the dataset in accordance with the ranked list ofanalytical calculations.
 17. The database management system of claim 11,wherein the operations further include: executing each analyticalcalculation within the list of analytical calculations on the dataset toproduce a list of result scores, wherein the list of result scoresincludes a result score for each analytical calculation.
 18. Thedatabase management system of claim 17, wherein the operations furtherinclude: updating the ranking of the list of analytical calculationsbased on the list of result scores.
 19. The database management systemof claim 11, wherein the analytical calculation characteristic furtherincludes: a type of calculation to be performed by the analyticalcalculation, wherein the type includes one or more of a linear operationand a square root operation.
 20. The database management system of claim11, wherein the dataset includes a data confidence value.